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ABSTRACT 

This report, prepared by the Delinquency and School 
Environments Program of the Center for Social Organization of 
Schools, presents a set of standards for implementing Program 
Development Evaluation ( PDE ) , a method for strengthening school 
organizational and research effectiveness. This report supplements an 
earlier account of the Program Development Evaluation ( PDE ) method 
for developing and evaluating programs by providing more explicit 
guidance in the creation and assessment of implementation and 
evaluation plans. Imperative, essential, and desirable standards for 
various elements of plans created using this method are described. 
Consultants, researchers, and program developers may apply these 
standards in determining whether or not they are actually 
implementing the PDE method, and these standards are useful in 
training workers fo use the method. (Author/PN) 
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Trie Center 



The Center for Social Organization of Schools has two primary objectives: 
to develop a scientific knowledge of how schools affect their students, and 
to us.e this knowledge to develop better school practices and organization. 

The Center works through three, research programs to achieve its objectives. 

. The School Organization Program investigates how school and classroom organiza- 
tion affects student learning and other outcomes. Current studies focus on 
parental involvement, microcomputers, use of time in schools, cooperative 
learning,, and other organizational factors. The E ducation and Work Program 
examines the relationship between schooling and students' later-life occupational 
and educational success. Current projects include studies of the competencies 

' required in the workplace, the sources of training and experience that, lead to 
employment, college students' major field choices, and employment of urban 
minority youth. The Delinquency and School Environments Program researches 
the problem of crime\ violence, vandalism, and disorder in schools and the 
role that schools play in delinquency. Ongoing studies address the need to 
develop a strong theory of delinquent behavior while examining school effects 
on delinquency and evaluating delinquency prevention programs in and outside 
of schools. 

The Center also supports a Fellowships in Education Research program that 
provides opportunities for talented young researchers to conduct and publish 
significant research and encourages th^ participation .of women and minorities 
in research on education. 

This report, prepared by the Delinquency and School Environments Program, 
presents a. set of standards for implementing Program Development Evaluation 
(?DE), a method .for strengthening school organizational and research ■ 
effectiveness . - 
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Standards for Program Development Evaluation Plans 



Abstract 

This xaport supplements an earlier account of the Program Development 
Evaluation (PDE) method for developing and evaluating programs by pro- 
viding more explicit guidance in the creation and assessment of imple- 
mentation and evaluation plans. Imperative, essential, and desirable 
standards for various elements of plans created using this method are 
described. Consultants, researchers, and program developers may apply 
these standards in determining whether or not they are actually imple- 
menting the PDE method, and these standards are useful in training work- 
ers to use the method. 



This document spells out in f 
greater detail than we have else- 
where the standards for implementing 
the Program Development Evaluation 
(PDE) method in practice. These , 
standards have a dual purpose: 
(a) they provide guidance 0 for tha 
researcher-consultant using the PDE 
method to conduct research . while 
stimulating organizational develop- 
ment, and (b) they form a basis for 
the assessment of a nlan to create 
or study change. The application of 
these standards assumes a basic 
knowledge of the PDE method. A 
description of that method (Gott- 
fredson, 1982) should be read before 
attempting to use this document. 

PDE is a complex set of interven - 
t ions : it is a method for accom- 
plishing the goal of increasing the 
usefulness of programs and research 
on them. The PDE method ic intended 
to. strengthen research and interven- 
tions to solve practical problems 
according to a specific theory of 
organizational change and develop- 
ment wh'.le doing useful research. 
This tLeory implies that several 
objectives must be met if organiza- 
tional and research effectiveness 
are to be improved. These objec- 
tives include: (a) increasing the 
extent to which theories are used in 
developing programs", (b) increasing 



the use of careful, detailed, and 
feasible plans in developing pro- 
grams, (c) increasing the use of 
val id., information in program deci- 
sion making, (d) increasing the 
relevance of information in program 
development and decision making, and 
(e) increasing the extent to which 
project implementers adopt the fore- 
going objectives (and indeed the 
theory underlying PDE) as their own. 

All of the standards described in 
this document are aimed at one or 
more of these objectives. For exam- 
ple, we call here for clear state- 
ments of what outcomes project 
implementers expect to observe as 
short- and long-term effects of 
their efforts, and we call fpr the 
provision of information that ena- 
bles the implementer to confirm or 
disconfirm those expectations. 
These steps are intended to accom- 
plish the objective of increasing 
the perceived relevance and use of 
information. 

These standards, although linked 
to a theory of organizational 
research and development, have mul- 
tiple additional sources. The first 
of these is. the principles of 
research design: Useful research 
must be concerned with validity, 
efficiency, robustness, cost, and 



theory. Tbo second is the 
psycl )logy of the behavior of indi- 
viduals and groups: Some of these 
standards are intended to build what 
is known of human mo r iv at ion and 
effective perf o*~vnaure into plans to 
manage an ot denization change and 
research effort. The third source 
is our experience in developing and 
implementing the PDE method in col- 
laboration with a number of organi- 
zations. 

These standards make clear that 
PDE is a highly complex method. Is 
there any way to simplify this pro- 
cess? We do not think so. Social' 
programs aio not only widely per- 
ceived as ineffective, but by and 
large they are ineffective. Evalua- 
tion research is widely perceived as 
irrelevant and useless, and by and 
large it _is irrelevant and useless. 
There is no more justification in 
assuming that complex human problems 
that have heretofore defied solu- 
tion — and many of which are regarded 
as intractable— will yield to simple 
approaches to solution than there is 
in assuming that it is easy to fly 
to the moon. The scientists, engi- 
neers, and managers who conducted ^ 
the moon missions developed elabo- 
rate plans with clearly specified 
theoretical rationales and multiple 
checkpoints with a high degree of 
redundancy in systems to accomplish 
the difficult task of going to the 
moon. We make no apology for the 
complexity of the method for which 
we provide standards here. Our firm 
belief is that what may here appear 
complex is in fact a great simplifi- 
cation and sy stematizatipn of the 
process required to solve human 
problems. If this method calls for 
more talent and investment of 
resources than are currently availa- 
ble, then more talent and more 
resources will b!e required to solve 
many of the world's problems. 



T ypes of Standards 

Various applications of the PDE 
method will embody its techniques to 
different degrees. But no one 
should be confused about whether PDE 
is actually being implemented, or 
whether the name alone is being 
applied to a set of activities that 
resemble PDE only slightly. To 
decrease ambiguity, we specify sev- 
eral kinds of standards: impera- 
tive > essential , and desirable. In 
addition, selected issues that are 
not really "standards" are men- 
tioned. 

Imperative 

An imperative standard must be 
present in all applications of PDE. 
If an imperative standard is not , 
met, the application is not follow- 
ing the PDE method. These are stan- 
dards that we judge to be such an 
important part of the method that 
they must' be present in all applica- 
tions of the method. 

Essential 

An essential standard must be 
present in every complete applica- 
tion of the PDE method. Although 
judged not to be so central to the 
method that they must always be 
adhered to, if an essential standard 
is overlooked or ignored the result- 
ing defect will weaken the applica- 
tion of the PDE method. 

Desirable 

A desirable standard is one that 
should be met if possible. Some 
desirable standards call for 
research methods or resources that 
are beyond the resources of some 
organizations and some consultants. 
Others may be applicable only in 
certain instances. As a general 
matter, desirable standards should 
be attended to and met as feasible ' 
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and applicable; the greater the 
extent to which these standards are, 
achieved, the more effective and 
useful the research and development 
project will be. 

Other Issues 

Some other issues that should be 
considered in the development of any 
PDE activity are discussed at the 
end of several sections. PDE is 
value laden, and ethical social 
scientists will want to consider the 
ways their activities influence 
human welfare or the political con- 
text of the project, and their 
proper role in relation to the 
client or clients. 

These standards do not stand 
alone; they are intended to supple- 
ment other standards, including the 
American Psychological Association, 
American Educational Research Asso- 
ciation, and National Council on 
Measurement in Education (1974) 
Standards for Educational and Psy - 
chological Tests » the American Psy- 
chological Association Ethical Prin - 
ciples (1981), Standards for 
Providers of Psychological Services 
(1977), and Ethical Standards in the 
Conduct of Research With Human Par- 
ticipants (1973), and the Evaluation 
Research Society (1982) Standards 
for Program Evaluation . In contrast 
to those statements, which provide 
broad, guidance on ethical issues, 
the present standards are intended 
to specify in concrete detail what 
is required to implement one spe- 
cific type of intervention with 
organizations: Program Development 
Evaluation. 

, Problem Statements 

The Process of Data-Guided Problem 
Definition 

- Imperative : Realism . PDE is a 
pragmatic exercise. Grandiose goals 



and unachievable aims are not 
"problems" in the language of PDE. 
The broad mission of an agency to 
f ight poverty or improve education 
consequently may not be stated as a 
goal in a PDE plan. Concrete prob- 
lems for which a resolution is con- 
ceivable are stated. If nothing can 
conceivably be done about n problem 
by a project, that is not. a problem 
the project is addressing. 

Essential: Needs assessment . 
Every problem statement should be 
based on some form of needs assess- 
ment; available evidence is used in 
performing this needs assessment • 
If hard data are not initially 
available, a vehicle for developing 
them must be specified. Needs 
assessment must be guided by data 
about the problems an organization 
is experiencing, its current level 
of functioning, or evidence about a 
d is cr epan cy be tw<* en pe r f ormance and 
needed or de sired performance • 
Accordingly, basing problem state- 
ments on epidemiological studies , 
opinion polls, surveys, archival or 
production records and so forth is 
desirable. The researcher must 
exercise considerable judgment in 
determining how thorough an initial 
needs assessment shall be • 

Essential: Diverge nt views . 
Divergent views should be considered 
in specifying problems. First, not 
all members of an organization share 
the same perspective on organiza- 
tional needs, and not all persons 
affected by the organization will 
share the organization's mananage- 
ment's views of needs. In addition 
to divergent values and perspec- 
tives, the views or opinions of any 
one. person or group may" be objec- 
tively incorrect. A careful needs 
assessment therefore involves con- 
sultation with a broad range of per- 
sons likely to be involved in or 
affected by the project, and it 
requires an open discussion of pri- 



orities. Second, the participation 
of affected parties in the develop- 
ment of plans may be expected to 
increase the extent to which members 
of an organization adopt the per- 
spective and practices of effective 
organization development and 
research embodied in the theory 
underlying PDE. 

To make the assessment of the 
degree to which this standard has 
been met possible, records of who 
has participated in plan development 
are helpful. 

Desirable: Normative informa- 
tion . The use of normative informa- 
tion to aid in the interpretation of 
the data is desirable. For example, 
knowledge that 70% of the parents of 
children in a school approve of the 
school's discipline policy is most 
useful if accompanied by information 
that it is typical that 85% of 
parents approve of school discipline 
pol icies . 

Issues for the Researcher and Imple - 
ment er 

Several issues will arise in most 
PDE projects; each of these should 
be considered by the researcher as 
he or she develops the project. 

Issue: Value judgments . Virtu- 
ally all prioritization of goals 
involves value judgments. It is 
useful if the judgments „and assump- 
tions involved are described and 
labeled as such, together with a 
discussion of competing priorities 
and values . 

Issue: Stakeholder identifica- 
tion . Clients, users, and stake- 
holders in the research should be 
clearly identified. Often .a PDE 
project involves multiple clients, 
and this may present challenging 
ethical and practical issues for the 
researcher. Often the priorities of 
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alternative clients may differ or be 
competing. Participants in a PDE 
project should know what they car 
expect of the researcher. 

Issue: Redundancy of services . 
In determining problems to address 
in any project, consideration should 
be given to the range of services or 
programs being directed at problems. 
Even if alcoholism is a big problem 
in a community, a community that 
already has a vigorous alcoholism 
prevention and treatment program may 
have greater needs in other areas. 

Issue: Who identifies the prob - 
lem? Whether or not the "client" 
sees the problems selected as a 
problem is important. In general, 
when applying the PDE method the 
"client" defines the problems to be 
addressed by the project. In many, 
if not most, applications of PDE 
there will be multiple clients; not 
all of these will share the proximal 
client's definitions. In many, if 
not most, interventions with organi- 
zations the client's self-identified 
"problem" is incorrect from the per- 
spective of the outside consultant, 
or the request for assistance may 
either seem inappropriate or be a 
request for an intervention other 
than the most useful one. Research- 
ers will generally want to distin- 
guish between-their diagnoses of an 
organization's "problems" and those 
identified by organizational managers 
or clients; and researchers will 
generally want to regard all diag- 
noses as tentative. Researchers 
should constructively confront the' 
client with their perceptions. 
Researchers must decide whether they 
can be helpful or can ethically pro- 
cede with a project. 

Issue: Human welfare . A primary 
aim of researchers using the PDE 
method is to promote human welfare. 
Each researcher should carefully 
attend to the ethical issues 
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involved in pursuing any particular 
project in light of this general 
aim. 

Issue: Who is the client ? To 
avoid misunderstandings | researchers 
using the PDE method should create 
an explicit understanding of ' who the 
primary client is , and who the sec- 
ondary clients are. 

Standards for Written Problem State - 
ments 

The following standards should be 
clearly observable in written docu- 
ments produced using the PDE method: 

Imperative: Distinct problem and 
theory / Application of the PDE 
method requires a separation of 
desired outcomes from ideas about 
their causes. Written problem 
statements must not confound cause 
with effect, or theory with problem. 
For example, "seventh grade Iowa 
test scores are ten points too low 
in Testemup Junior High School" is a 
problem statement. But, "little 
student participation in curricular 
decision making" is not. The reason 
for this is that the first statement 
describes a concrete problem, 
whereas the second statement 
describes one of many possible theo- 
ries to explain a number of school 
outcomes. Statements of the second 
kind do not follow the PDE method, 
because the method separates causal 
states from their expected effects 
to enable tests of these causal 
hypotheses . 

Imperative : Quant if ication . 
Problems are stated in behavioral 
and quantifiable terms, and quanti- 
tative indicators of the level of 
the problem are described and avail"" 
able. If quantitative indicators 
are not initially available, a plan 
to develop them is made and exe- 
cuted. 



Essential : Cone ret enesH . 
Written statements of problems 
should be specific and concrete 
rather than vague, abstract, or gen- 
eral • 

Goal Statements 

General Standards for Goal State - 
ments 

The following standards apply to 
the development and writing of goal 
statements according to the PDE 
method: 

Imperative: Operational meas - 
ures . The measurement operations to 
assess each goal are specified. 
These operations can actually be 
performed, i.e., it is possible to 
obtain information to measure each 
goal. 

Imperative: Expected effects . 
The magnitude of the effect sought 
or anticipated is explicitly stated. 
Without a statement of how large an 
effect is expected, this expectation 
can not be concretely compared with 
information collected. In addition, 
information about expected effect 
size helps create an experimental 
design with sufficient statistical 
power. Statements of expected 
effect size should be clear (e.g., 
"one-half standard deviation," "five 
percent," or "from the 33rd to 50th 
percentile." Such statements are 
not vague or obscure; for example, 
words like "significantly," "note- 
worthy," "discernable" are avoided. 

Imperative : Timing of effects . 
Goal statements clearly specify when 
their attainment is to :e expected.. 
This (a) promotes realism in plan 
development, and (b) aids in evalua- 
tion design by specifying when out- 
comes should be measured. 

Imperative : Scope . The target 
group or scope of measurement is 
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specified. Thio means that the 
target population is defined in 
terms of personal > geographic, or 
community characteristics, and the 
limits or boundaries of the project 
are therefore also defined. "Ameri- 
can Black teenage unemployment rates 
are double those of whites" is an 
acceptable problem statement only if 
the project is targeted at all Black 
teenagers in the entire United 
States. 

Essential : Completeness * Goal 
statements are complete: a goal is 
stated for each problem. 

Issue: Honesty . Goal statements 
are honest; little is to be gained 
by exaggerated, unrealistic goals, 
or statements of goals that are in 
fact not being sought. The goals 
should be realistically attainable 
and of moderate difficulty. Clearly 
some judgment is required in apply- 
ing this exhortation. 

Research Design Standards for Goal 
Statements 

Imperative: Internal validity . 
A research design that will rule out 
all plausible rival causal interpre- 
tations is used if possible. The 
basis for inference about project 
effectiveness in bringing about each 
goal is specified. A number of 
methods can be used to achieve 
internal validity (Cook & Campbell, 
1979). 

Essential: Opportunism . Obvious 
measures of goals are not over- 
looked. For example, in a project 
aimed at increasing academic 
achievenent in a school system with 
a standardized testing program, test 
scores derived from that program are 
not overlooked. A variety of archi- 
val records and other kinds of evi- 
dence can often be capitalized on in 
designing the research (Webb, Camp- 
bell, Schwartz, & Sechrest, 1966). 



Essential: Power . The 
experimental or quasiexperimental 
design for the evaluation is statis- 
tically powerful (sec Cohen, 1970). 

Eflflential: Fall-bnck design s. 
Fall-back designs are anticipated. 
(It is often necessary to anticipate 
fall-back designs while at the same 
time guarding against the possibil- 
ity that project implementers or 
research staff will regard the 
fall-backs as "good enough.") 

Desirable: Multiple measures . 
Multiple measures are available for 
each goal . 

Desirable: Decision making 
rules . Estimates of the disutility 
of Type I and Type II errors are 
made. These estimates should guide 
project decision making based on 
evaluation feedback by determining 
cutoff values for statistical tests. 
The risk of falsely concluding that 
evidence of effectiveness is lacking 
for an effective program can often 
be so great that conventional sta- 
tistical significance levels make 
little sense, especially in pilot 
work where sample sizes may be 
small • 

Theor y Statements 

In using the PDE method, theory 
statements make the rationale under- 
girding a project explicit. 

Generatin g the Theory 

c 

Much confusion exists about 1 the 
ways theories are created. Some 
commentators on theories in the 
behavioral science argue that few if 
any so-called theories really have 
the characteristics of sound theo- 
ries, but philosophers of science 
continue to argue about what those 
characteristics are, about the mean- 
ing of "cause, 11 and so forth* In 
using the PDE method, the most 



important test of a theory is, M Ie 
it useful?" A problem facing the 
researcher is the surfacing and 
explication of the often implicit 
thcorieo held by practitioners, or 
of helping the implementer choose, 
create, and specify a theory of 
action using the ideas of others... 
Accordingly, a central task for the 
researcher is to help state a theory 
in propositional form so that it may 
guide intervention and be tested. 

Useful theories have many ori- 
gin?: sometimes thay come from the 
systematic work of scientists who 
have stated theories or systematic 
perspectives; sometimes they are 
created by the synthetic activity of 
practitioners who aim to make sense 
of their experiences. There exists 
no single best source of theories. 

The PDE method requires the 
statement of theories in proposi- 
tional form where that is possible. 
But no one should be inappropriately 
seduced by the formalism of proposi- 
tional statements. Such statements 
may serve to prematurely crystal ize 
thinking and actually cause harm if 
taken too seriously (Kaplan, 1964). 
Formalism and propositional form are 
no substitutes for good ideas, and 
human behavior is often too complex 
and our knowledge too limited to 
allow for sensible modeling or for- 
mal ization. 

As a practical matter, however, 
unless a theory specifies statements 
such as, "If we do this, then that 
will happen, " it can neither guide 
the choice of interventions in a 
project nor be tested. Accordingly, 
the PDE method requires the develop- 
ment of such propositional state- 
ments, and encourages active experi- 
mentation to test the statements in 
practical application (cf. Cook & 
Campbell, 1979, Chap. 1). ,A good^ 
theory has utility. It guides 
action, suggests new approaches, and 
suggests tests of its propositions 



or implications. A theory, no mat- 
ter how formal, that leads nowhere 
ia useless. Examples of proposi- 
tions developed using the PDE method 
should resemble the fbllowing: "If 
80% of children are imunized using 
vaccine X, the transmisoion of 
disease Y will cease." "If 65% of 
the households in a neighborhood 
actively watch for crime and report 
suspicious events to the police, the 
property crime rate will drop." "If 
each instance of disruptive behavior 
in the classroom is consequated with 
a soft verbal reprimand, the inci- 
dence of disruptive behavior will 
decline." "Employment of youths 
decreases the probability that they 
will be arrested for a crime." 

A good theory explains the pheno- 
menon of concern. Therefore, each 
action theory should be scrutinized 
to determine if it corresponds with 
what is known or believed to be true 
more generally. Does this theory 
make sense? Does some general prin- 
ciple or set of principles account 
for a large number of diverse obser- 
vations? 

Many useful theories are proba- 
bilistic. That is, their proposi- 
tions are not of ,the form* "if X 
then Y," but rather of the form, "X 
increases (decreases) the probabil- 
ity of Y." Here we assert the use- 
fulness of what Cook and Campbell 
(1979) call molar theories. In gen- 
eral, we do not know enough about a" 
problem to specify all its causes or 
the conditions under which they 
operate. Nothing is to be gained by 
refraining from constructive theor- 
izing simply because some epistimol- 
ogies (operational dogma, essential- 
ism) would label our language 
invalid. Much is to be gained from 
the elaboration of partial or molar 
theories accompanied by tests of 
those partial theories, in practical 
appl ication. 



'3 



To be put to practical use, a 
theory must suggest operations to 
manipulate the causal variables, and ^ 
to measure the outcomes. Developing 
a program and testing a theory 
involve, a cycle of activities to 
sharpen up definitions of theoreti- 
cal terms over time as the theory is 
tested and experience is gained. Of 
'course, the statement of theoretical 
terms will become clearer as this 
process occurs. And., because any 
particular operation or measure is 
ambiguous, multiple measures and 
multiple operations are to be pre- 
ferred when possible. 

■ q 

Standards for Developing Theory 
Statements 

Although useful theories may have 
diverse origins, all theories that 
are useful for problem solving and 
organizational development using the 
PDE method have important features 
in cogmon. The "following standards 
are helpful in assessing and writing 
theories. The standards stated 
below provide guidance by spelling 
out some characteristics that a 
testable theory will have. 

Imperative: Theory of action . 
The stated theory must form a real- 
istic base for project actions. 
Grand theories that imply interven- 
tions beyond the resources or scope 
of a project are not useful, and 
theories that are too narrow, for the 
w resources or scope of a project are 
of diminished utility. An action 
theory must be translatable into 
\ statements of objectives and inter- 
ventions. The theory must be such 
that it does suggest concrete 
actions that may be taken N to solve 
the problem. 

Imperative: Scope I The phenome- 
non the theory attempts to explain 
must be clearly stated. This state- 
ment determines the scope of the 
theory . Issues such as what the 



target population for .the project 
will be (which were resolved in the 
problem definition stage) help' 
define the scope. 

Imperative: Internal consis- 
tency . The theory- is internally 
consistent — it coheres as a state- 
ment, and no part contradicts or is 
implied to be false by any other 
part. - 

Imperative : Propos itional form . 
The theory is stated in.proposi- 
tional form, so that it can be 
empirically tested by the operation 
of the project. Statements such as 
"Other things being equal, the more 
the A the more the B 11 ' or "If X then 
Y" are required. 

Essential: "Causal order . Theory^ 
statements should clearly indicate a 
hypothesized causal ordering. For 
example, "A leads to B" is a more 
useful ^theory statement than "A is"' 
linked to /B." A diagram illustrat- 
ing the causal order should, either 
be drawn, or be easy to draw from 
the theory* statement. 

Linear causal orderings do not 
always capture the nature of beha- 
vioral processes. Sophisticated 
theories sometimes specify cyclical 
processes or reciprocal causation. 
At the same time, most projects take 
action primarily at one point- in: the 
process. The point of intervention 
then specifies a causal ordering 
underlying the project activities^. 
The location of the intervention in 
this chicken-and-egg cycle must be 
clear. o 

Essential: Correspondence with 
evidence and other theory . A useful 
theory corresponds with all or most 
x>f the evidence available about a * 
phenomenon: It fits with the facts 
insofar, as they are known or under- 
stood. The validation of theories 
is a. complex business, but some 



theories are clearly invalid given 
an informed perspective on the phe- 
nomenon. An example .may be helpful. 
Consider two alternative theories 
about worker productivity. A human 
relations perspective might imply 
that increasing worker satisfaction 
wil 1 increase productivity . But 
this theory has problems :* First the 
evidence (Brayfield { & Crockett, 
1955) shows that job satisfaction is 
weakly associated with performance. 
Second, the general notion that 
reinforcement establishes and main- 
tains behavior is much more solidly 
established (Pritchard, Leonard, 
VonBergen, & Kirk, i976 ; Yukl & Lat- 
ham, 1975). Third, theories (e.g.* 
Porter & Lawler, 1968) exist which 
specify why job satisfaction may 
expected to be only weakly linked 
with performance. One might be able 
to increase male assembly line work- 
ers** job satisfaction by having 
attractive coctail waitresses in 
brief costumes serve beer on a con- 
tinuous basis and do nothing (or 
more likely decrease) job perfor- 
mance. On the other Hand, making 
the administration^ of desired 
rewards contingent on increased per- 
formance, as is implied by an alter- 
native theory, appears more promis- 
ing. In chosing a theory to guide a 
project, the available evidence 
about & the validity of alternative 
theoretical perspectives should be 
used. 

This standard is essential rather 
than imperative because it is some- 
times impossible or impractical to 
convince project . implementers of the 
invalidity of their personal theo- 
ries. In such cases, it is in the 
spirit of the PDE method to subject 
the theory to empirical test. The 
researcher must, however, attempt to 
bring as much knowledge as possible 
to bear on the assessment of the 
theory undergirding an activity so 
that unnecessary research is avoided 
and each project stands on the 
shoulders of others. 



Knowledge of the problem, and 
theories of the problem that have 
been subjected to some prior tests, 
are taken' into account in developing 
the theory "of action for a project. 

Issue: Probabilistic statements . 
Determinism is often misunderstood. 
If appropriate , propositions should 
be stated in probabilistic form. 

Standards for Written Theory State- 
ments 

The foregoing provide guidance 
for the development of theory state- 
ments. The following additional^ 
standards pertain to the written" 
product of this process. 

Imperative: Coherence of the 
plan . The theory must be a theory s 
of the problem the project seeks to 
solve. The links between the prob- 
lem, theory, and interventions 
developed should be clear and 
straightforward. Coherence can be 
completely assessed only by review- 
ing problems, theory^, objectives, 
and interventions together. Fur- 
thermore, every major project compo- 
nent, or intervention, should be 
directly traceable to an element of 
the project's theory. , 

Imperative: Clear narrative . A 
theory must be stated in narrative 
form* in crisp, clear, and unambigu- 
ous prose. This means that "expres- 
sion is direct and interpretable , 
that separate ideas are expressed 
distinctly, that each idea is devel- 
oped sufficiently that readers can 
understand them, and that vague 
wording is avoided. For example, 
"weakened' bonds to the social order 11 
is by itself an imperfect theory 
statement. f, The weaker the bonds to 
the social order (i.e., the less the 
attachment of youths with their 
parents and teachers, and the less 
youths perceive that educational 
attainment is essential to obtaining 



valued personal goals) the more the 
delinquency 11 is a better theory^ 
statement. Even the latter state- 
ment should be elaborated considera- 
bly. 

Essential : Parsimony * The guid- 
ance of the Bishop of Ockham should 
be followed: Theory statements 
should be no more complex- than 
necessary to explain the phenomenon 
which is the object of the theory; 
(This principle is sometimes 
described with vivid imagery as 
Occam's (sic) razor.) A more parsi- 
monious statement is not only more 
aesthetically pleasing, but also 
easier to test (Kaplan, 1964). 
(( 

Essential: Generality. A sequi- 
ter of the foregoing standard is the 
requirement of generality. A gen- 
eral theory is superior to specific 
propositions because it is more 
robust: it explains more, and sug- 
gests more potential interventions. 
For example, a theory that students 
do not come to school because no one 
notifies the school when they hang 
out at the arcade is less general 
than a theory that students will 
engage in whimsical activities 
unless they are watched and their 
behavior is consequated. The latter 
theory is more useful because it 
suggests a broad array of community 
control interventions, whereas the. 
former suggests only activity^ 
directed at the arcade attending 
behavior. 

Essential: Comprehensi veness . 
The theory should be as complete an 
explanation as is possible at the 
time of the phenomenon which is its 
object. In general, a. project that 
ignores an important known cause of 
a problem is likely to be ineffec- 
tive in solving it. Thus, a cancer 
prevention project aimed at reducing 
exposure only to one of the known 
cancer risk factors may be expected 
to be less effective than a project 
aimed at reducing exposure to many 



or most of the known cancer risk 
factors. The more important the 
risk factor (or more influential the 
cause) the weaker a project that 
ignores it. Thus, a cancer preven- 
tion project that ignored the risks 
caused by cigarette smoking would be 
less effective than a project that 
ignored only exposure to sunlight. 

Objectives 

Relation of Objectives to Other PDE 
Elements 

Objectives and interventions put 
the theory of action to work. 
Objectives are the causal states 
that, a project's theory says must 
occur for the desired outcoae to 
occur. Interventions are the activ- 
ities performed to bring about those 
causal states (or objectives). In a 
treatment program designed to manage 
hypertension, for example,. a theory 
may state that if salt intake is 
reduced, blood pressure will then be 
reduced as well. A physician pre-, 
scribes a low sodjLum diet and gives 
the patient a low sodium cookbook 
and menu plan with instructions for 
the patient rigidly to adhere to the 
diet, eating only the prescribed 
foods. The physician proscribes all 
other foods. In this example, the 
patient's blood pressure is a meas- 
ure of the goal, the sodium content 
of the diet consumed is a measure of 
the objective, and the preparation 
and consumption of the prescribed^ 
food (and only that food) are indi- 
cators that the intervention is 
being performed. A physiological 
measure assesses goal attainment, a 
chemical measure assesses the objec- 
tive, and a behavioral measure 
assesses the intervention. (Imple- 
mentation standards are discussed in 
a subsequent section: These stan- 
dards are PDE's tools for assessing 
the faithfulness with which inter- 
ventions are implemented.) ' '* 




A second example is useful: A 
police chief theorizes that property 
crime in £ certain suburban neigh- 
borhood could be lowered if more 
surveillance were possible* Of the 
many possible ways to increase sur- 
veillance, the chief decides to use 
members of the community in a "Crime - 
Watch" program. The chief also 
theorizes, however, that community 
members do not currently perceive 
the importance of their roles in 
neighborhood control, and that they 
wil 1 only participate in surveil- 
lance if this citizen role is made 
more salient. In' this example, the 
property crime rate in the neighbor- 
hood/measures the goal, the ratio of 
daily reports of suspicious events 
in progress to crimes reported might 
be used to measure one of. the objec- 
tives (surveillance), and the pro- 
portion of neighborhood r< ' < .nts 
signing an agreement to pa*, . cipate 
in the program and posting a "Crime 
Watch" sticker on their front doors 
might be used to assess the other 
objective (sal iency of the citizen 
role). Finally, to implement the 
program the police chief would 
require each pol ice of f icer to 
explain the Crime Watch program to 
the householder on each call for 
assistance and on every crime inves- 
tigation, and would prepare a news 
release for the local media. 
Records of officer activity on 
visits to residents' homes arid 
counts of television and newspaper 
news stories might be used to docu- 
ment the degree of implementation. 

The foregoing examples clarify 
the relation between theory, objec- 
tives, and interventions; and they 
illustrate the value of treating 
each of these elements of the PDE 
process separately. Objectives are 
states hypothesized to cause the 
desired outcome.* Interventions are 
activities undertaken to create the 
causal conditions necessary . We 
want light '(goal) so we get tungsten 
hot (objective) by applying 



electrical current to a circuit in 
which a tungsten bridge is the 
point of great resistance ( interven- 
tion). If hot tungsten did not 
glow, or if the light switch were 
not thrown, the bulb would not 
light. 

Standards for Written Objectives 

Many of the standards for written 
objective statements are the same as 
standards for written goal state- 
ments. Because each is important, 
however, the common standards are 
repeated here (although in abbrevi- 
ated form) to emphasize their impor- 
tance. 

Imperative : Causal states . Art 
objective is never an action taken. 
Actions are interventions , not 
objectives. Maintaining the dis- 
tinction between actions (interven- 
tions) and causal states (objec- 
tives) is imperative because of the 
problematical link between the - 
intervention and the objective. A 
group of people living in a repres- 
sive society may theorize that 
greater freedom will bring greater 
happiness, but the group would be 
mistaken to confuse the revolution 
(action) with freedom (objective). 
Often revolutions succeed merely in 
changing the boot on the peoples 1 
necks. Objectives are intermediate 
outcomes; they lie between the 
action and the goal. 

Imperative: Theoretical rele- 
vance . Objectives are operational 
measures of the action theory's 
causal variables. 

Imperative: Operational meas- 
ures . The measurement operations to 
assess each objective are -specif ied. 
These operations can actually be 
perf ormed, "i.e. , it is possible to 
obtain information to measure each 
objective. 
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im perative; gg Bgcted ef f ects .- 
The magnitude of the effect sought 
or anticipated is explicitly stated. 

Im perative: Timing gj effects . 
Objective statements clearly specify 
when their attainment is to be 
expected. 

, Im perative: Scope . The target 
group or scope of measurement is 
specified. < 

Essential: Completeness . Each 
hypothetical causal variable- is 
operational ized by a measure of an 
objective. This standard is essen- 
tial rather than imperative, because 
as a practical matter all causal 
conditions may not be specifiable at 
the * time, and because for many pur- 
poses a complete specification is 
helpful but not imperative. For 
example, a child who does not under- 
' stand the principles of electricity 
may know nothing of the mechanism 
thrbugh which the light switch oper- 
ates. For the child's purposes it 
is important to know that throwing 
the switch usually causes light. 
The child's control over the envi- 
ronment, however, falters if the 
fuse is blown or the light bulb is 
burned out or missing. 



Objective 
statements are realistic. That is, 



Issue; Realism. 



they can plausibly be attained and 
are of moderate difficul-ty. Clearly 
some judgment is required in apply- 
ing this standard. 

Research D esign Standards for Objec- 
n'yp Statem ents 

T ^r^wp; Inte rnal validity. 
A' research design that will rule out 
all plausible rival causal interpre- 
tations is used if possible. The 
basis for inference about project 
effectiveness in bringing about each 
objective is specified. 



Essential? Opportunism . Obvious 
measures of objectives are not, over- 
looked. . i 

Essential: Power. The experi- 
mental or qua si experimental design 
for the evaluation is statistically 
powerful . 

Essential: Good es timators. < 
Messures called for by the research 
design are efficient and unbiased. 

" Essential: Fall-back designs. 
Fall-back designs are anticipated. 

desirable: Multiple m easures. 
Multiple measures are available for 
each objective. 

Desirable: Decision making 
. rules . Estimates of the disutility 
of Type I and Type II errors are 
made. These estimates should guide 
project decision making based on 
evaluation feedback by determining^ 
cutoff values for statistical tests. 

Interventions 

The standards for interventions 
flow from considerations of 
(a) strength, (b) integrity, and 
(c) feasibility. See Sechrest, 
West, Phillips, Redner, and Yeaton 
(1979) for a discussion of the first 
two issues. In our adaptation 
"strength" refers to the theoretical 
plausibility of an intervention—the 
links between what is done and the 
objectives that must be met to^ 
accomplish a goal. "Integrity 
refers to the faithfulness with 
which an intervention is carried 
out. "Feasibility!' refers to the 
likelihood that the intervention can 
be implemented as intended. A 
strong intervention is one that is 
implied by a plausible theory, or 
one that is a close replication of 
another intervention known to work. 
An intervention with integrity is 
one that is carried out as antici- 
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pated. A feasible intervention is 
one the organization can see ways to 
adopt and implement* 

In describing standards for the 
choice of interventions we will 
f ocu 6 pr imar il y on s t r eng t h . 
Because of their special importance, 
considerations of feasibility and 
integrity are treated in more depth 
in separate sections of these stan- 
dards on forcefield analysis and 
implementation plans. 

The Choice of Interventions — 
Strength 

Imperative: Target population . 
Interventions must be focused on the 
target population identified in the 
problem statement. 

- Imperative: Target objective ♦ . 
Interventions must be targeted at 
one or more objectives. 

Essential : Comprehensiveness . 
Interventions should be aimed at 
bringing about all (or the most 
important) of the objectives. 

Essential: Proximal priority . 
In choosing interventions, actions 
taken to most directly manipulate or 
bring about the causal state (objec- 
tives) the theory implies must be 
present must be given precedence 
over actions that may be. expected to 
influence that causal state more 
remotely. The example used else- 
where (Gottf redsoii, 1982) of inter- 
ventions aimed at cholera can be put 
to another use in illustrating this 
requirement. When a person has the 
disease (problem), he. or she is in 
danger of dying due to dehydration 
and electrolyte imbalance (theory). 
A direct intervention (and 'therefore 
a strong and preferred intervention) 
is the administration of a "soup" 
containing water and electrolytes to, 
rehydrate the person and restore 
electrolyte balance (objective). 



But when a cholera epidemic occurs 
in a community (problem) the treat- 
ment of victims is only an indirect 
intewention (and therefore weaker 
and less preferred intervention). 
Because cholera is transmitted from 
infected persons to others through 
contaminated drinking water (theory 
of transmission) action taken to 
break transmission by chlorinating 
wells and encouraging the populace 
to boil their water are more direct 
(and therefore stronger and more 
preferred) interventions . Finally, * 
if cholera is endemic in a community 
(problem) this is usually because 
environmental sanitation is ~poor 
(theory) and actions taken to 
<- improve sanitation by constructing 
and encouraging the use of safe 
water supplies and sanitary latrines 
are direct and appropriate interven- 
tions because they make the water 
supply /resistant to contamination by 
the cholora micro-organism (objec- 
tive). Although other considera- 
tions in addition to the proximal 
nature of the intervention (such as/ 
the mission, forseeable lifespan, or 
budget of the intervention effort, 
or community sentiment) contribute 
to the wise choice of interventions, 
proximality is a major considera- 
tion. 

Essential: Duration, frequency, 
dosage, and timing . An essential 
element of the choice of interven- 
tions is the determination of their 
appropriate duration, frequency, 
dosage, and timing. 

„ It is self-evident that the 
descr iption of nutritional interven- 
tions designed to promote the health 
of a population must involve state- 
ments about the amounts of various 
substances to be consumed with what 
frequency and at what stages in the 7 
life cycle. Too much calcium is / 
toxic, and the body's requirement 
for this mineral is not constant 
over the life span. Too little cal- 

/ 
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cium during infancy, childhood, and 
adolescence is harmful, but calcium 
def iciency * is rarely a problem in 
adulthood and indeed overconsuuiption 
of milk products is probably more 
widespread than deficiency among 
adults. 

For an intervention designed to 
break the transmition of cholera 
during an epidemic, . the intervention 
of water-boiling would, specify that 
water is to heated during the epi- 
demic for three minutes (duration), 
always (frequency), to 100 degrees 
Centigrade (dosage), before use 
(timing). For an intervention 
designed to help unemployed persons 
get jobs (Azrin & Besalel, 1980) it 
might be specified that persons 
attend a job club until they are 
employed (duration), every day (fre- 
quency), all day (dosage), if they 
are adults (timing). - 

Any of these elements — duration, 
frequency, dosage, and timing — that 
may be ambiguous must be described 
in an intervention statement. 

Essential: Use of knowledge . 
Interventions should make use of 
knowledge about other successful and 
unsuccessful interventions. Often, 
where other interventions are known 
to have worked in the past to 
achieve an objective, a project's 
-intervention should begin with as 
exact a replication of that inter- 
vention as possible, introducing 
changes only on an experimental 
basis. If project implementers can 
justify major deviations from inter- 
ventions known to have worked in the 
past on the basis of presumed 
increased efficiency, effectiveness, 
or because of more limited 
resources , these deviations should 
be made on an experimental basis 
only . Interventions unsuccessful in 
prior attempts would be replicated 
only where ideas for ways to 
strengthen or modify them to cause 
them to be productive are available. 



Choice of Interventions — "Feasibility 

Imperative : Perceived f eas ibil - 
itv . Project implementers must per- 
ceive that a chosen intervention can 
be adopted and implemented. This 
means that they must perceive that 
resistance to any innovation can be 
overcome and that adequate resources" 
to achieve implementation arc avail-^ 
able. This perception should be 
supported by a careful force-field 
analysis, and by an assessment of 
human, real, and financial 
resources. 

Integrity of Interventions 

Imperative : Act ions taken . An 
intervention statement must specify 
what actions are to be taken to 
achieve an objective. Such state- - 
ments must be clear and unambiguous . 
A statement such as "vocational 
counseling 11 is unclear and is ambi- 
guous. "Vocational counseling using 
Parsons' (1909) methods," or "Simu- 
lated vocational counseling using 
Holland's (1979) Self-Directed 
Search ." are clearer. 

Essential : Manuals . The clear- 
est specification of the actions 
taken in an intervention is a well 
worked out blueprint for behavior or 
a manual. The Job Club Counselor 's 
Manual (Azrin & Besalel, 1980) is a 
good example of such a document. 
This manual must detail exactly what 
must be done, and include all 
instructions and forms necessary to 
implement and monitor the execution 
of the required actions. A complete 
manual also describes goals, objec- 
tives, theory, and required 
resources. 

Essential : Staffing .- The number 
and kinds of staff required to 
implement the intervention must be 
specified. A job description for 
each category of staff is required; 
Competencies that qualify the work- 
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ers to implement the intervention 
must also be specified. For exam- 
ple, an environmental sanitation 
campaign may require cement masons 
and community workers; a bridge con- 
struction project engineers and 
steelworkers ; and a hospital , physi- 
cians y nurses, laboratory techni- ~~ 
cians, and. a psychological techni- 
cian trained in the treatment of 
tracheotomy addiction. The general 
principle is that staff with the 
competencies to take the actions 
desired will be needed, and a care- 
ful plan anticipates this need, 

Force-field Analysis ( FFA) 

Force-field analysis is a techni- 
que that is useful in developing 
feasible plans because it makes 
explicit the perceived obstacles to 
change — forces that maintain a sta- 
tus quo — and the resources for over- 
coming these forces or obstacles. 
Force-field analysis is a movable 
component of the PDE method. It 
often provides useful information in 
choosing a problem, in deciding 
whether or not a theory of action 
will be palatable or politically 
acceptable, in determining which 
objectives are most manipulable, in 
choosing among alternative interven- 
tions aimed at the same objec- 
tive(s), in developing strategies to 
achieve adoption of an intervention, 
in developing plans to achieve 
faithful implementation of an inter- 
vention, and in devising plans for 
evaluation of project activities. 

Four principles provide guidance 
in the application of FFA. 

Imperative: Perceived feasibil- 
ity . The individual or group to 
conduct the project must perceive 
that a path exists to change the 
status quo — to change practices or 
implement new ones. Put another 
way, the individual or group must 
expect that the available resources 



are sufficient to overcome iach 
perceived obstacle. If a path is 
perceived and there is no uncer- 
tainty about it, formally performing 
a FFA may be unnecessary. When 
there is uncertainty, conducting a 
FFA is imperative. 

Essential : Completeness . All 
perceived obstacles must be listed 
and systematically considered. This 
allows a rational and systematic 
consideration of resources for over- 
coming obstacles, turning them into 
resources, or dasigning plans to go 
abound an obstacle or render it 
ijrrelevant. Unless this task is 
approached systematically, useful 
strategies are often overlooked 
-because workers despair of attempt- 
ing innovation in view of what may 
be perceived as intractable obsta- . 
cles. 

Essential : Qpen-mindedness . FFA 
is performed before promising ideas 
are abandoned. 

Pes ir able : Variety . Although no 
complete list of the kinds of fprces 
that may be present is possible , it 
is possible to list some of the 
kinds of obstacles and resources 
that should be considered. This 
list includes money, attitudes, 
. talents, norms, structural arrange- 
ments, contracts or agreements, 
power or influence, management prac- 
tices, rewards, punishments, and 
t ime • 

Strategies: The Management Plans 

According to the PDE method, a 
^strategy is a plan for getting the 
\work done. A complete strategy has 
three kinds of elements. The first 
of these are the cr itical benchmarks 
required to bring about a change in 
the status quo and to maintain that 
change. These critical benchmarks 
are changes ih the field of forces 
that determine what the status quo 
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will be: When critical benchmarks 
are attained, the status quo 
changes. The second kind of element 
in a strategy are the implementation- 
standards that, guide the work of 
project implementers by specifying 
what constitutes faithful implemen- 
tation of the interventions. These 
standards provide concrete day-to- 
day guidance and worker reinforce- 
ment, and they allow managers to 
monitor the quality of the interven- 
tions. The third kind of element 
are the tasks required to bring 
about critical benchmarks and to 
achieve implementation standards. 
These tasks specify who will do 
what, when. Examples of critical 
benchmarks, implementation stan- 
dards, j and tasks may be found in 
Gottfredson (1982). 

General Standards for Strategies 

Imperat ive : Commun icab ilitv . 
The plan to manage change and imple- 
ment interventions must be communi- 
cable. A narrative statement of 
what the plan involves should be 
prepared to provide coherence to the 
lists of details usually developed. 

Imperative: Inclusion of bench- . 
marks . Every management plan spells 
out^ the critical benchmarks that 
must be achieved to get an innova- 
tion adopted and maintained. 

Imperative: Inclusion of imple- . 
mentation standards . Every manage- 
ment plan spells out the standards 
that must be met to faithfully 
implement the intervention^ ) , or 
has as a critical benchmark the com- 
pletion of such implementation stan- 
dards to be applied at a later time. . 

Essential: Distinctions among 
elements of the plan . Tasks must be 
distinguished from critical bench- 
marks and implementation standards. 
Each of these elements in management 
plans serves a distinct purpose: 
Tasks provide day-to-day guidance to 



workers and their accomplishment is 
rewarding; critical benchmarks sig- 
nal changes in the force-field and 
are necessary to manage innovation; 
implementation standards indicate 
the faithfulness with which an 
intervention is be ing implemented. 

Desirable: Task statements . A 
complete enumeration of tasks is 
usually desirable. When the nature 
of the tasks is ambiguous, or . any 
uncertainty about who is to do what 
by when Exists, an enumeration of 
. tasks is essential . 

Critical Benchmarks (CB's) 

Imperative: Consistency with 
FFA . Critical benchmarks, should 
clearly follow from a force-field 
analysis. (Critical benchmarks are 
to FFA what objectives are to 
theory.) 

Imperative : Concreteness . CB's 
must be statements of specific, 
observable changes in driving or 
restraining forces. 

Essential: Dates . Each critical 
benchmark must specify a date by 
which it is expected to have been 
met. 

Essential : Completeness . The 
set of CB's listed must be complete 
in the sense that all or the most 
important of the restraining forces 
are coped with (overcome, converted, 
or rendered impotent ) . 

Im plementation Standards (IS's) 

0 IS's should be the result of a 
careful consideration of the charac- 
teristics of interventions implied 
by the project's theory of action. 
They should also use information 
from previous interventions about 
the characteristics of those inter- 
ventions necessary for effective- 
ness. 




Imperative: Provisions for moni- 
toring . Implementation standards 
state the procedures used to monitor 
interventions, and .the frequency, 
timing, and sampling procedures for 
this monitoring. IS's must be spe- 
cific, observable, dated, and recur- 
rently monitored. 

Imperative : Observable . IS's 
must be observable conditions or 
behaviors. They do not specify the 
"potential 11 to do anything, they 
specify what is done. 

Essential : Competencies and 
time . If the strategy involves the 
use of staff with special skills or 
competencies IS's must be created to 
determine whether staff actually 
have these skills or competencies." 
IS's to specify and monitor the 
extent to whicji sufficient time is 
being devoted to the required activ- 
ities are necessary. 

Essential : Comprehensiveness ♦ 
'The plan must contain IS's for each 
element of the intervention. 

Task Statements 

Imperative: Who, what, when . A 
task statement must specify who does 
what by when . x 

Essential: Detail * Task state- 
ments are sufficiently detailed that 
workers know what to do. 

Management Plan for Evaluation 

Evpiuation is a project component 
like any other. Accordingly, a 
clear and detailed management plan 
for evaluation activities is 
required* Most of what has been 
discussed already therefore, appl ies~v 
here. A few additional special 
standards, and some that require 
emphasis, are outlined here. 

Imperative: Management plan . A 



detailed management plan for evalua- 
tion, developed from a careful 
forcefield analysis, is imperative. 

Essential : Res pons ibilities . 
The foregoing management plan fully 
lays out responsibilities for col- 
lecting and analyzing data. 

Essential : Realism . The sche- 
duling of inf ormation e flow is real- - 
ist ic. 

Essential : Management and 
research needs . The management and 
research needs are met by the data 
collection schedule. 

Essential: Information quality * 
Implementation standards are devel- 
oped to specify the quality of 
information required. Such stan- 
dards will involve, for example, 
sampling procedures, response rates, 
construct validity of the measures, 
and timing of randomization. 

Standards for Information Feedback 

In the PDE method information is 
a tool of project development. 
Information is used to reexamine 
each element of a program plan in 
the light of evidence to determine 
whether or not changes or improve- 
ments may be useful. Elements may 
be added, deleted, or modified if 
information is feedback . to implemen- 
ters and used by them. The follow- 
ing standards provide guidance in 
deciding what information to gather 
and in feeding back information to 
proj ect implementers : 

General Standards for Feedback 

Imperative: Expected vs . 
observed . Information - is directly 
linked to conditions the project 
implementers expect to observe. 
Thus, information about goal attain- 
ment is presented in conjunction 
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with information about initial 
expectations. Similarly, informa- 
tion derived from monitoring imple- 
mentation is presented in conjunc- 
tion with information about the 
implementation standards. Discre- 
pancies between expected and 
observed outcomes are highlighted. 
Presenting information in this fash- 
ion is^intended to make its per- 
ceived relevancy high, and to pro- 
vide motivation through the tension 
inherent in a discrepancy between 
what is expected and what is 
observed to be happening. 

Imperative : Collaboration . The 
determination of what information to 
collect, the interpretation of the 
information, and decisions about 
project modification are activities 
in which project implementers and 
researchers mutually influence each 
other. This mutual influence is 
intended to keep the research rele- 
vant to the project and the project 
relevant to the research. 

The Process of Deliverin g Feedback 

Imperative: Timeliness . Infor- 
mation must be provided before it is 
useless for making project adjust- 
ments. Information is most useful 
if present at the time a decision is 
to be made, such as during natural 
planning occasions. It is essential 
that project implementers identify 
such occasions. Obviously, feedback 
of information about tasks and cri- 
tical benchmarks, and implementation 
standards may often be needed more 
frequently or sooner than informa- 
tion about objectives or goals. 
Frequently project implementers 
themselves must gather and. interpret 
short-term information to meet this 
information need. 

Im perative: Time to review 
information . Researchers and imple- 
menters must plan for and spend 
ample time together to review and 
interpret information. 



Imperat ive : Under s t andab il itv . 
To be used, information must be 
understood by project implementers. 
Because implementers differ greatly 
in their sophistication in using 
information, no uniform guidelines 
are possible. The following key 
ideas, however, are helpful: 
(a) Keep feedback as simple as pos- 
sible, (b) Avoid esoteric statis- 
tics in displays; translate complex 
results into plain English or graphs 
whenever necessary. (c) Make infor- 
mation self-interpreting if possible 
by, for Example, the use of norms or 
verbal interpretations. (d) Make 
discrepancies between expected and 
observed outcomes obvious (e.g., by 
using the\ metric of ' the expecteds to 
present information about observeds, 
and by presenting expecteds and 
pbserveds ^together). (e) Accompany 
tables or graphs by narrative inter- 
pretations, (f) Do hot provide too 
much information at once; highlight 
critical information. (fc) Use proj- 
ect decision-makers' categories and 
concepts. (h) Present information 
in ways tha^t can be understood by 
pr oj ect impl ementer s . 

Imperative: Unintended conse - 
quences . The possibility of side- 
effects or cither unintended conse- 
quences of an intervention or 
decision alwkys exists. Plans 
should be matfe to scan the environ- 
ment for sucl| unintended conse- 
quences and measure them as appro- 
priate. Information about 
unintended consequences is also fed- 
back to project implementers. 

Essential:! Constructive feed- 
back . Information should be pre- 
sented using a constructive, persua- 
sive style — using for example the 
« guidance provided by Hakel , Sorcher, 
Beer, and Mose'a (1982). Tact, pati- 
ence, honesty ,j concreteness, and 
helpful suggestions are needed. 
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Issue: Appearance of validity * 
In some circumstances information 
may appear to have more legitimacy 
if collected over extended periods 
of time, for large samples, etc. 
There is no necessary connection 
between the utility or validity of 
information and the extensiveness of 
efforts to collect it. 

Content of Information Feedback 

Imperative: Links between infor - 
mation and plan components * Three 
conditions are imperative in provid- 
ing information of relevance to 
project decisions: (a) Links 
between the information and project 
goals, objectives, or operations are 
explicated. (b) Information is 
directly aimed at isues raised by 
the project ; or it provides clear 
information about the project's 
theory, goals, objectives, critical 
benchmarks, or implementation . stan- 
dards, (c) Information highlights 
unintended consequences or important 
but unexpected observations* 

Essential: Informed observa - 
t ions . Researchers must provide 
information derived not only from 
formal data gathering activities, 
but also from their observations of 
project structure, communication, 
decision making, goals, and activi- 
ties; and from behavioral theory and 
research. These observations are 
linked to project goals, theory, 
objectives , interventions , critical 
benchmarks , and implementation stan- 
dards as appropriate • Observations 
of mismatches between project theory 
and interventions are especially 
important. 

Essential : Comprehensivness * 
Information is provided about all, 
or the most important of, a proj-. 
ect's elements. 



Ethical Concerns 

Several ethical issues may arise 
more frequently in the feedback pro- 
cess using the FDE method than in 
other kinds of research. Some of 
these are highlighted by the follow- 
ing issues: 

Issue: Welfare of subjects . The 
rights, privacy, and welfare of all 
individual subjects of research must 
be stringently safeguarded. 

Issue: Feedback to all stake- 
holders . Although not imperative 
when using the PDE method, it is 
often highly desirable (for both 
practical and ethical reasons) to 
provide relevant information to all 
stakeholders in an evaluation 
research project. The timing, con- 
tent, and relevance of information 
flow often raises important ethical 
issues discussed. elsewhere (Evalua- 
tion Research Society, 1982). 

Issue: Inappropriate use of 
information * Researchers must 
attempt to educate their clients in 
the appropriate use of information. 
Special cases of sensitive informa- 
tion require careful consideration 
in light of ethical standards elabo- 
rated elsewhere* Del iberate distor- 
tions of information or evaluator 
statements occur frequently — not 
everyone shares the ethical perspec- 
t ive s of re searchers • ( See Evalua- 
tion Research Society, 1982.) 

Issue: Limitations of evidence * 
Researchers must make the limita- 
tions of their research methods 
clear when presenting feedback. 

Issue: Trust * Effective pro- 
grams are more likely to be devel- 
oped when implementing organizations 
trust the researchers to bring both 
the good and the bad news. Often, .. 
however, research methods are neces- 
sarily so complex that project 



implementers can not possibly 
evaluate the research. They must 
therefore trust the researchers, 
thereby possibly becoming dependent. 
This complexity can raise important 
ethical issues, and at the very 
least implementers should be 
assisted in ^their attempts to get 
competent second opinions. 

Development 

0 The PDE method recognizes that 
projects change over time with 
experience. The PDE plans must 
therefore also change as the project 
develops. Each component of the 
plan may change: Problem, goals, 
theory, objectives, interventions, 
analyses of the forcefield, and so 



on. The following additional stan- 
dards are also required: 

Imperative: New information 
needs . When project decisions are 
made within the PDE framework, new 
information needs generally emerge. 
Steps must be taken to develop the 
information required to examine the 
consequences of project decisions. 

Imperative: Periodic review . 
Occasions must be created for the 
periodic review and revision of the 
entire plan. 

Imperative: Thorough review . 
Every part of the program and its 
evaluation should be reassessed in 
these periodic reviews. 
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